Computing Confidence Intervals for Common IR Measures

Confidence intervals quantify the uncertainty in an average and offer a robust alternative to hypothesis testing. We measure the performance of standard and bootstrapped confidence intervals on a number of common IR measures using several TREC and NTCIR collections. The performance of an interval is its empirical coverage of the estimated statistic. We find that both standard and bootstrapped intervals give excellent coverage for all measures except in situations of abysmal retrieval performance. We recommend using standard confidence intervals when statistical software is handy, and bootstrap percentile intervals as equivalent when no statistical libraries are available.